System and method for predictive curation, production infrastructure, and personal content assistant

ABSTRACT

Described herein are systems and methods that utilize data points, calendar entries, trends, behavioral patterns to predict and pre-emptively build digital and printable products with selected collections of images without the user&#39;s active participation. Based on analysis of a user&#39;s collections and online behaviors, the system may estimate types and volumes of potential media-centric products, and the resources needed for producing and distributing such media-centric products for a projected period of time. Embodiments of the system comprise a user interface that takes the form of a “virtual curator,” which is a graphical or animated persona for augmenting and managing interactions between the user and the system managing the user&#39;s stored media assets.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. application Ser. No.15/611,378, filed on Jun. 1, 2017, (issued as U.S. Pat. No. 11,429,832),which claims priority to U.S. provisional patent application No.62/344,761, filed Jun. 2, 2016; 62/344,764, filed Jun. 2, 2016; and62/344,770, filed Jun. 2, 2016. The disclosures of the above-referencedapplications are hereby incorporated by reference in their entireties.

BACKGROUND OF THE INVENTION

The present invention relates to providing services to a user based onanalysis of the user's preferences and behaviors. In particular, thepresent invention relates to providing and recommending services to auser based on analysis of information collected from the user's mediaassets (e.g., photograph, music and video collections), on-line, andsocial media activities.

With the proliferation of portable networked devices (e.g., smartphones, tablets, laptops, and connected digital cameras), the costsassociated with recording and storing multimedia assets areinsignificant. Consequently, users are accumulating vast collections ofstill images and video clips. The amount of a user's personal multimediaassets makes it difficult to identify and locate the more importantones. Also, because of a desire to document, recognize, and memorializeevery day and important life events, many users record images and videosas everyday events unfold. Typically, such multimedia content are firstcaptured in the internal memory of a device, which is then transferredto a hard drive of a personal computer, a networked storage device, aportable hard drive, a solid state memory, or remote cloud-basedstorage. Very often, the best multimedia content recorded at anindividual's important event is recorded by another attending the event,but who has neglected to share this content with the other attendees,despite being a close personal friend or a family member to theindividual. Therefore, a convenient and systematic way for locating,sharing, and using assets from multiple multimedia content collectionscan be very valuable.

Many users store personal content, such as images, in cloud-basedstorage services (e.g., DROPBOX™, GOOGLE® PHOTOS, AMAZON® CLOUD DRIVE),the actual capture device (e.g., a smartphone), a portable hard drive, apersonal computer, or on a social network (e.g., FACEBOOK®), or acombinations of these approaches. However, as mentioned above, all toooften when a user wishes to retrieve a specific image, to share an imagewith a friend or family member, or to use an image in a personalmedia-centric gift (e.g., a photo greeting card, photo calendar, photoalbum, or a digital movie or slideshow), he or she is unable to locatethe image in a timely and efficient manner.

Analysis of a user's media assets, such as photographs, or music andvideo collections, enables various commercial or social applications.Such applications are disclosed, for example, in (a) U.S. Pat. No.7,836,093, entitled “IMAGE RECORD TREND IDENTIFICATION FOR USERPROFILES” to Gobeyn et al.; (b) U.S. Pat. 8,910,071, entitled “IMAGEDISPLAY TABS FOR ACCESSING SOCIAL INFORMATION” to McIntyre et al., (c)U.S. Pat. No. 8,028.246 entitled “CONCIERGE SHOPPING ASSISTANT” also toMcIntyre et al., (d) U.S. Patent Application Publication 2009/0132264,entitled “MEDIA ASSET EVALUATION BASED ON SOCIAL RELATIONSHIPS” by Woodet al.; (e) U.S. Pat. No. 8,832,023, entitled “SYSTEM FOR MANAGINGDISTRIBUTED ASSETS AND METADATA” to Blomstedt et al.; and (f) U.S. Pat.No. 8,934,717, entitled “AUTOMATIC STORY CREATION USING SEMANTICCLASSIFIERS FOR DIGITAL ASSETS AND ASSOCIATED METADATA” to Newell et al.

In data mining, transaction histories (e.g., purchases, onlineactivities, social network interactions) have been used to derive usefulinformation about individual and group behaviors. A transaction recordis typically identified by a transaction identifier and records a set ofitems involved the transaction. This record format is called “marketbasket” style data, as it is similar to a listing of the contents of asupermarket shopping cart of an individual shopper. A transactionsdatabase contains a large set of transaction records. Data mining toolshave been developed for extracting frequently occurring groups of items(“itemsets”) from conventional transactions databases. There has beensome work in using data mining techniques in the image collectiondomain. For example, see U.S. Patent Application Publication2011/0072047 to Wang et al., entitled “INTEREST LEARNING FROM AN IMAGECOLLECTION FOR ADVERTISING;” see U.S. Pat. No. 6,598,054, entitled“SYSTEM AND METHOD FOR CLUSTERING DATA OBJECTS IN A COLLECTION,” toSchuetze et al.; and see U.S. Patent Application Publication2008/0275861 to Baluja et al., entitled “INFERRING USER INTERESTS.”

SUMMARY OF THE INVENTION

According to one embodiment of the present invention, a predictivecurator analyzes a user's media assets, transaction data, calendarentries, trends, behavioral patterns to predict and pre-emptively builddigital media-centric products using the user's collections of images,with minimal or no active participation by the user. The user'scollections of media assets may be retrieved from files on the user'sdevice, cloud-based photo library, or other libraries shared among otherindividuals and grouped into thematic products. Based on analysis of theuser's collections and on-line behaviors, the predictive curator mayestimate types and volumes of potential media-centric products, and theresources needed for producing and distributing such media-centricproducts for a projected period of time.

According to one embodiment of the present invention, the “virtualcurator” may take the form of a graphical or animated persona foraugmenting and managing interactions between the user and the systemmanaging the user's stored media assets. The virtual curator can assumeone of many personas, as appropriate, with each user. For example, thevirtual curator can be presented as an avatar-animated character in anicon, or as an icon that floats around the screen. The virtual curatorcan also interact with the user via text messaging, or audio messaging.

According to one embodiment of the present invention, the curatorperforms co-occurrence tagging to facilitate searching of the mediaassets. To provide seed terms for tagging, the curator can takeadvantage of knowledge embedded in social network comments, andindividuals and predominate objects recognized in the media assetsthemselves. The curator can also take advantage of co-occurrence termsin associated users' collections for matching and correlation.

The present invention is better understood upon consideration of thedetailed description below in conjunction with the accompanyingdrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an overview of a system 10 in which a predictive curator maybe implemented, in accordance with one embodiment of the presentinvention.

FIGS. 2 a-2 b illustrate a flow chart that illustrates the work flow ofa curator according to one embodiment of the present invention.

FIG. 3 is a flow chart showing how a persona may be selected for theuser, according to one embodiment of the present invention.

FIGS. 4 a, 4 b, 4 c and 4 d show image Content and contextualindications useful for learning a user profile, according to oneembodiment of the present invention.

FIGS. 5 a, 5 b, 5 c and 5 d show various examples of interaction betweenthe predictive curator and the user in the course of the curator's workflow, according to one embodiment of the present invention.

FIGS. 6 a, 6 b, 6 c and 6 d illustrate the tone of the questions thatwould be asked from two different personas, in according to oneembodiment of the present invention.

FIGS. 7 a, 7 b, 7 c and 7 d illustrate the curator persona may askquestions about an image that incorporate already recognized oridentified features in the image, in accordance with one embodiment ofthe present invention.

FIGS. 8 a, 8 b, 8 c and 8 d show different curator personas that thecurator may assume, according to one embodiment of the presentinvention.

FIGS. 9 a and 9 b show two forms that the curator may present to a userto customize a greeting card to be created, in accordance with oneembodiment of the present invention.

FIG. 10 a shows a look-up table that can be used by the curator to aprofile of the desired product, according to one embodiment of thepresent invention.

FIG. 10 b shows components of a greeting card that can be selected basedon the values assigned to one or more of the profile categories,according to one embodiment of the invention.

FIG. 11 elaborates steps 209-212 in the flow chart of FIGS. 2 a -b,according to one embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Whether for lack of interest, ability, creativity, imagination, time, orskill required, a user is often reluctant to manually search his or herimage collections. The present invention provides a “predictive curator”which is an application program with a graphical user interface (“GUI”)that communicates between the user and image analysis and productselection application programs. The curator organizes and manages theuser's audio and visual media assets (e.g., the user's collections ofpictures, music, movies, and TV shows) based on learning about the userfrom these assets and other data (e.g., demographic information andpersonal style preferences). The curator may also identify opportunitiesfor creating compelling media-based products.

FIG. 1 is an overview of a system 10 in which a predictive curator maybe implemented, in accordance with one embodiment of the presentinvention. As shown in FIG. 1 , the predictive curator of the presentinvention may interact with a user via the user's computing devices. InFIG. 1 , for example, the user's computing devices may include mobiledevice 100 (e.g., a smartphone or a tablet computer) and desktop device20, which communicate with each other through a wireless link 90 (e.g.,BLUETOOTH™ or WiFi). On desktop device 20, the user may interact withthe curator through GUI 110, which is displayed on graphical display 30,mouse 80, and speaker 70. The user also has access to peripheral deviceson desktop device 20, such as camera 40, memory card and/or disk readers50, 60 and output devices 130, 140, 150 and 160. Output devices 130,140, 150 and 160 may be, for example, printers of various types,including printers suitable for printing photographs 120 and 3-Dprinters. In FIG. 1 , desktop devices also has access to router 170which provides access to a wide area computer network (e.g., theinternet), through which the curator, running on desktop device 20 ormobile device 100, may access and make use of remote computing resource10 (e.g., remote server 180). System 10 can be embodied as a personalcomputer system or as a retail photo kiosk system.

Significantly, according to one embodiment of the present invention, theuser is provided a timely presentation of important images and potentialphoto-centric products incorporating the user's own relevant multimediaassets, without triggering any of the user's real or imagined privacyconcerns. In some instances, such products may be created using outputdevices 130, 140, 150 and 160, as illustrated in FIG. 1 . The presentinvention also provides the user with a creative, expressive opportunityto recognize and memorialize important life events by transforming whatwould otherwise be an involved craft project into a personalizedshopping experience.

FIGS. 2 a-b show a flow chart that illustrates the work flow of acurator according to one embodiment of the present invention. As shownin FIG. 2 a, at step 201, a user initiates the curator to run on acomputational device. In some embodiments, the curator may be downloadedfrom a vendor's server on the internet. When the curator is initiatedfor the first time, at step 202, the curator presents to the user aservice agreement to which the user must consent. FIG. 5 a shows thecurator, assuming a persona “Veronica” interacts by voice with the userto introduce itself and to obtain permission to proceed as a curator forthe user's media assets, in accordance with one embodiment of thepresent invention. Thereafter, the user provides the curator permissionto access the user's contacts, social network accounts, relationships,event calendar, time and location data, and collected multimedia assets.FIG. 5 b shows that the curator, after taking an inventory of the user'smedia assets locally and on social media, requests by voice message theuser's permission to process the media assets and social interactions onsocial media, in accordance with one embodiment of the presentinvention.

At step 203, the curator processes the information provided by the userto create metadata for its use. Such metadata may include, for example,metadata that encompasses the multimedia assets, significant individualsand events. The processing that creates such cross-referenced or indexedmetadata are provided in greater details below. FIG. 5 c shows thecurator, having processed the user's multimedia assets and social mediainteractions, reports its finding by voice message to the user,according to one embodiment of the present invention.

At step 204, the curator stores the created metadata for easy access.The data may be stored locally, remotely or both (e.g., on desktopdevice 20 or remote computing resource 10 of FIG. 1 ). At this time,i.e., at step 204, the curator may interact with the user to obtainfurther information about the user and to create a profile based on theinformation retrieved from the user or from the metadata derived fromprocessing the user's information obtained in step 202. The curator mayassign the user to a persona category, as discussed in further detailbelow.

At step 205, the user selects a desired level of interaction with thecurator, which may include, for example, the frequency of interactionand one or more modalities of interaction (e.g., text messages, email,voice or audio messages, over GUI or through an avatar). At step 206,the user may authorize the curator to interact with one or more of thesignificant individuals identified in step 203 for interaction (e.g.,sharing of the user's multimedia assets). At step 207, based on theuser's approval at step 206, the curator may contact the significantindividuals to share the multimedia assets, or to introduce thepredictive curator application program. At step 208, the curator thenreturns to step 201.

After the initial contact of steps 202-208, initiation of the curator atstep 201 leads the curator to step 209. At steps 209-210, the curatorprocesses any newly acquired multimedia assets and metadata in themanner already described with respect to steps 202-203. At step 211,based on processing the user's media assets, the curator may prepare oneor more customized media-centric products to be offered to the user orone or more of the significant individuals. (Each customizedmedia-centric product, e.g., a photo album, incorporates one or more ofthe user's media assets.) Some of the methods used by the curator topredict the content, the relevant events and the potential recipient orrecipients for these products are disclosed in further detailed below.If the curator, at step 212, determines an opportunity to present one ofthe customized media-centric products is realized (e.g., a week before arecipient's birthday), at step 213, the curator may create and present avirtual version of the customized media-centric product to the usertogether with one or more suggested recipients and the event or occasionon which to present the customized media-centric product. (A virtualversion of the product is a computer-model of the product, so as toallow the user to examine, visualize, edit or otherwise modify theproduct through the curator's GUI.) FIG. 5 d shows an example in whichthe curator recognizes an opportunity to share a media-centric product(e.g., a photo album) with one of the user's significant individuals(e.g., the user's sister). In FIG. 5 d, the user seeks the user'sconsent to share that media-centric product, according to one embodimentof the present invention.

In one embodiment, steps 209-212 can be further described with referenceto FIG. 11 . The assets referenced in steps 209 and steps 210 from FIG.2 a are collectively shown as assets 2005 in FIG. 11 . Analyzing assets2005 by metadata generation process 2010 yields metadata 2015, which maybe represented according to the industry-standard data model RDF(Resource Description Framework), and which may be stored in database2020. Metadata generation is further described below. Databases designedfor storing RDF data, such as database 2020, are known in the industryas triplestores. The triplestore 2020 constructs the appropriateauxiliary indices for efficient data retrieval. Step 211 of FIG. 2 aitself can be described in further detail. For example, the process ofidentifying and ranking important media assets, known as “storygeneration” begins with story generation component 2040, which may bebased on metadata 2015 from triplestore 2020. In one embodiment, a storyis merely a particular grouping and prioritization of a set of assets.Different algorithms may be used to generate different story types. Thisembodiment may support a variety of story generation algorithms. Themost basic algorithm groups assets temporally, but in a hierarchicalmanner, so that the grouping reflects an event structure. For example,assets captured over a multi-day vacation may correspond to a single“super-event” and assets captured each day might correspond to separate“events.” The assets belonging to an event may be further classifiedaccording to separate subevents, which correspond to differentactivities that occur within the event. Subevents may be furtherorganized based upon degrees of similarity between adjacent images.

A second algorithm for story generation organizes assets thematically,by identifying a common theme. Theme identification may be accomplished,for example, using frequent item set mining (described subsequently inmore detail). Other algorithms may use alternative grouping mechanisms.Each algorithm will further prioritize each asset based uponcharacteristics of the asset relative to the story type.

The operation of Step 211 of FIG. 2 a (e.g., initiation of storygeneration component 2040) is precipitated by a trigger in trigger set2030. The trigger may be as simple as the receipt of new assets, assuggested by the workflow of FIGS. 2 a -b. However, step 211 may beactivated by other triggers, including temporal triggers (e.g.,someone's birthday is coming up), location triggers (the user isdetected to be in proximity to a certain location), system triggers(e.g., a promotion on a certain type of product is to be run), andvarious combinations of these and other trigger classes.

The stories generated by story generation component 2040 are typically,although not necessarily, developed with a particular product class inmind. The process that associates a story with a particular productclass is referred to as “curation,” which preferably presents the storyto the user in a visual form. A completely generic story may simplyrepresent a grouping and prioritizing of a set of assets. The highestpriority assets can then be shown to the user. Stories developed with aparticular product in mind may only make sense when visualized inconjunction with that realized product. For example, one possibleproduct is a twelve-month calendar. In that case, the story grouping andprioritization may be much more specific (e.g., the story is presentedin the form of exactly twelve product groupings, and for each grouping,a single asset is given the highest priority). Such a story is bestvisualized as the product for which it was intended, e.g., as acalendar. Other possible product classes include collages andphotobooks. Some product classes correspond to a single surface, such asa collage or photo mug;

other product classes have multiple surfaces, such as photobooks orcalendars. Story generation component 2040 of FIG. 11 uses productprofiles 2035 as input to story algorithms when generating storiesintended for specific product classes. Story generation may be furtherrefined for a given user by reference to user profile 2050. A userprofile may be referenced by a story generation algorithm to refine thegrouping or prioritization process. For example, if a given user isknown to like cats, then pictures of cats may receive a higher prioritythan pictures without cats. More generally knowledge of the places, thepeople, the activities and the things important to a given user canresult in pictures portraying those places, people, activities or thingsreceiving a higher priority than they would otherwise.

In one embodiment, story generation component 2040 may generate multiplecandidate stories 2045 visualized by multiple candidate media-centricproducts. While these candidate media-centric products may be directlypresented in visual form to the user, the possible candidatemedia-centric products may be first screened using goodness assessor2060. Goodness assessor 2060 operates by combining user profile 2050with business rules 2055 to filter the set of candidate media-centricproducts 2045 to result in a set of good media-centric products 2060. Ifa good media-centric product 2060 is identified under test 212 of FIG. 2b, for example, the processor creates and presents to the user acorresponding virtual version of the custom media-centric product, asshown in step 213 of FIG. 2 b.

At step 214, the user authorizes the curator to send the customizedmedia-centric product, after editing or modifying by the user, ifdesired, to the intended recipients. Alternatively, the user may beoffered alternative customized media-centric products for selection.

The additional information obtained by the curators in steps 209-211 and213-24 may be used at step 215 to update the user's assigned personacategory. The work flow completes at step 216.

Along with recording video, sound, text, and still images, a capturedevice includes metadata associated with its sensors. For example, acellular phone, tablet, or digital camera may include location data froma Global Positioning Systems (GPS) or cell tower triangulation,orientation and inertia sensors, or a digital compass, accurateautomatically set and updated time and date, temperature and humidity,and data regarding peripheral devices with wireless or wired connectionsto the capture device. Such peripheral devices may include, for example,wearable health and physical activity devices that monitor physicalactivity levels and heart rates, remote cameras and microphones, andautonomous aerial drones that are directed from and connected to thecapture device and which are capable of recording and transmitting videoand high resolution still images in real time. In addition, data fromaerial drones which operate autonomously, such as data from GPS,altitude sensors, inertial, orientation, and directional sensors, andgyroscopes, may also recordable. Some recorded metadata, such as GPSdata, can be linked with information about the user (e.g., latitude andlongitude information may be linked to a street address or maplocation).

According to one embodiment of the present invention, by examining otheruser information, the curator may associate or “tag” such a streetaddress or map location with, for example, the user's home, arecreational property, a favorite camp ground, a trout fishing spot, ora hiking or mountain biking trail. In turn, such additional informationprovides further context regarding the user's life style and interests.After having been identified, analyzed, and cataloged, the curator mayrelate the metadata with the user's account, relationships, andsignificant events. Links to the media assets may be created, so as tofacilitate access when needed (e.g., for sharing or to produce acustomized media product). These tasks may be performed withoutrequiring the user to establish a new or redundant storage for theirmedia assets or personal content, thereby alleviating any privacyconcern relating to their availability at a new or additional location.

In some embodiments, processing of the media assets (e.g., acquiring therecorded metadata, deriving the metadata from images of objects, peopleor scenes) may be carried out by temporarily uploading the media assetsto a remote processor. In addition, to create virtual and physicalproducts, multimedia assets may be uploaded as needed from relatedparticipating users for incorporation. These images and the relatedmedia-centric products may be stored for a fixed time period to allowadditional orders to be placed.

To allow the media assets to be more searchable and to be able torecognize and correlate related events or individuals, “semantic tags”may be applied to the media assets. Such a technique is available fromexisting tools, such as Microsoft Corporation's “Computer Vision API” or“Clarifai”. Clarifai creates a set of semantic labels or tags for agiven image or video asset, with each label assigned an associatedconfidence score which indicates the probability that the tag isrelevant to the image or video frame tagged. Clarifai uses a model thatis based on deep learning, such that a feedback mechanism enables thesystem to improve its model over time.

In one embodiment, the tags assigned to an image indicate a degree ofquality or “interestingness” of an image. Machine learning techniquesmay be used to correlate tags or sets of tags against referencesgenerated from actually assessed aesthetic quality or the level ofgeneral interestingness of a picture. This data may be further refinedby personalizing the model for a given user or demographic. In additionto associating tags with individual images as done by e.g., Clarifai,tags may be associated with events. The set of tags associated with theimages in an event may be processed to identify frequent tags, unusualtags or stop words using conventional document analysis techniques. Fromsuch processing, a profile for the event may be developed. Such aprofile may include, for example, a set of tags characterizing the eventand an assessment of the significance of the event. A user profile maybe generated for each candidate set of images, and a metric may bedeveloped to compare between user profiles.

In general, tags should have an ontological structure (e.g. as chess isa type of board game, the tag ‘chess’ should be understood to match thetag ‘board game’). Tools such as WordNet and its direct or inheritedhypernyms may be used to form an ontology. An ontology may also be builtupon the concept expansion method, which is based upon domain-specificknowledge.

Using ground-truth data (i.e., actually tested data), we can generateprofiles for predetermined event semantic classes, such as birthdayparties or school recitals. From either a candidate single image or aset of images with a computed profile, one may algorithmically determinewhich of the predetermined semantic event classes best matches thecandidate image or image set. Such determination enabling one or moresemantic event classes to the candidate set to be assigned, togetherwith an associated confidence score.

When classifying a set of images using this approach, the set of tagsfor a set of images is obtained from the union of the tags associatedwith the individual images, after possibly filtering out unusual ornon-descriptive tags. The tags may be further weighted, based upon theirfrequency of occurrence and their associated confidence scores. Theweighted set may be compared against reference profiles. Whenclassifying a single image, the frequency of occurrence of each tag isone, but the confidence scores for the individual tags can still be usedfor profile comparison.

Once event tags are generated for images of a collection—including tagsfor hierarchical event classes—the tag sets be used (a) for identifyingthematic groupings of events; (b) as a component of the thematicgrouping algorithm, (e.g., as one of the features considered by afrequent itemset mining algorithm, see below); and (c) for a method inwhich thematic groupings of events is obtained, such as performed in (b)but without considering tags, followed by processing each theme toobtain tags for each thematic group independently. The resulting tagsassociated with an individual thematic group can then be used todetermine significance or otherwise score a set of thematic groupings.This approach allows one to make a collage from a set of thematic groupswithout knowledge of which thematic grouping or groupings are mostimportant. Thematic groupings may be ranked, for example, by consideringthe uniqueness of the tags, as measured either against the individual'scollection, or a subset of the collection, or as measured against otherusers. Alternatively, one can define an expert system based on prioriknowledge of tag profiles of higher value themes. Over time, the tagsets or profiles that are of greatest interest to a particular user maybe learned. For example, for a given user, the curator may learn thatdog or pet-centric images or events are of higher import than otherimages or events. For another user, the curator may learn that abstracttags such as “love” are of higher value.

Learning may be accomplished by examining usage patterns. For example,if a user posts an album on a social media network in which there aresubsets of images that are captured at different times, one may inferthat the subsets represent thematic groupings. By running a thematicgrouping algorithm on the subsets, the relative importance of thethematic groupings may be obtained. For example, an album including babyimages with a different pose each month can be so tagged to identify athematic grouping. Such tags may facilitate searches (e.g., to findsimilar albums taken by others in a social network), and for relevancereasons. For example, if one person shows affinity to pictures of aparticular theme, (e.g., frequently “likes” pictures of thematic group),the person may be alerted to additional pictures of the theme as theyare added to the album. In this application, the semantics of the actualtags are irrelevant; the value of the approach is realized by groupingof similar (i.e., highly matching) tag sets or profiles.

Tags may be used to help identify event boundaries, augments the methodsthat use visual similarity and temporal event clustering, as visualsimilarity metrics are sometimes inaccurate. A collection of images maycontain pictures captured by different individuals at differentlocations for overlapping events. The use of semantic labels may provideimproved event segmentation, including the ability to support parallelevents. Tags are also useful in identifying recurring events based onidentifying sets of images having the same tag sets. Such recurringevents may be prioritized based upon the tags. For example, picturesthat are tagged to have been taken at a park in July become moresignificant if there are similar such pictures taken every year. Theimportance of certain tags may be determined, for example, withreference to aggregated results from other users (e.g., if a largenumber of people find birthday pictures to be important, then birthdaypictures for a particular user can be deemed important). Of course, theactual weight relevant to a user may be learned over time based on userfeedback. Alternatively, when the tags associated with an event areunusual relative to other events, then that may indicate the event issignificant. Of course, the importance of any event may be derived frommultiple indicia. Identification of significant or important event maybe can be further enhanced by filtering out infrequently occurring butinsignificant events. Such filtering may be developed based on aggregatebehavior extracted from multiple collections. For example, filtering mayreduce the effects of a few pictures of washing machines taken from anisolated appliance shopping event. Conversely one may have a couple ofpictures of a school bus. Such pictures, perhaps representing the firstday of school, may be significant. Filters may be developed using expertknowledge, or obtained from the aggregate (e.g., one may create filtersby generating tag sets for the pictures people post on social medianetworks).

The tags may be used to suggest to a user commemorative photographicproducts. For example, when it is learned that a user takes pictures ofpumpkins every fall, the curator may automatically put together acollage of the consumer's best pumpkin shots over the years.Alternatively, the curator may remind people to take certain types ofpictures relevant to a particular location or around a particular date.For example, the curator may remind people to take their annual pumpkinshoot in October. In some instances, while high-level classification(e.g., a “pumpkin shoot”) is typically difficult to infer, the reminderneed only take the form of displaying example pictures from past events.

The curator may also suggest usage for an image or a set of images basedon the set of associated tags. In one embodiment, images that have beenposted by the user on social media may be referenced for tagging otherimages. Suppose the images that have been posted to social media havebeen labeled “second”. The “second collection” may include otherpreviously captured or unprocessed images. Access to the user's socialmedia account allows the curator to examine associated metadata such as“likes”, emotions, comments and other metadata that have been associatedwith those images posted on the social media. After tags have beenassigned to the newly captured or otherwise unprocessed images, thecurator may compare the generated tags with the tags in the secondcollection to identify similarly tagged images in the second collection.If the curator finds that a given image has a set of tags thatcorrespond to tags of a set of previously shared images, and if thepreviously shared images tend to be shared with certain social mediagroups, the curator may suggest sharing of the present image with thesame social media groups. Further, if the previously shared images tendto be well-received (“liked”) by a particular subset of the social mediagroups, the curator will highlight the present image for sharing inparticular with that subset of social media groups (or score it high oncorresponding newsfeeds). The curator may detect a usage pattern of thepreviously shared images and suggest the present image for the sameusage.

Of course, images need not be confined to a single collection. Thecurator may create different image collections that correspond todifferent individuals or groups of individuals (“affinity groups”). Bycorrelating tags in each collection, the curator may suggest usage ofimages or activities based on their affinity group or upcoming eventsidentified in the affinity group.

In addition to tagging based on identification in a thematic algorithm(e.g., using frequent itemset mining, described below), groups may alsobe formed simply based on one or two features, such as place orlocation. In combination with ontological reasoning and with referenceto other auxiliary data, such groups enable one to infer characteristicsassociated with a place. For example, when the tags on pictures from aknown location type (e.g., a zoo) are found to closely correlate with aset of closely matching tags from another location, one can infer thatthe other location may be of the same type (i.e., also a zoo).

Although semantic tags are discussed above primarily in conjunction withtagging events, the same method is equally applicable to classifyingactivities. For example, when certain tags are typically associated witha particular type of activity (e.g., Little League games), one may applythe same tags to another set of similar pictures (i.e., as also a LittleLeague game). The set being characterized might be from an event, butcould also be some other collection (e.g., a parent saving the bestpictures from a number of Little League games).

To accomplish its tasks, the curator may also use rule sets, dynamiclearning, or combinations of these and other suitable techniques. Forexample, the curator may use keyword co-occurrence techniques forcorrelation among the media assets.

In some embodiments, comparing the user's image content and associatedmetadata to a set of pre-determined user profiles (“personas”), thecurator may assign the user to a persona or persona category, so thatfunctions associated with the assigned persona may be performed inconjunction with the user, as already mentioned above with respect tothe flow chart of FIGS. 2 a -b. FIG. 3 is a flow chart showing how apersona may be selected for the user, according to one embodiment of thepresent invention. As shown in FIG. 3 , at step 301, the curator usesthe stored cross-referenced metadata (e.g., the metadata created andstored at steps 203-204 of FIGS. 2 a-b ) to draw up a questionnaire toquery to the user. Based on the user's responses to the questionnaireand other information available to the curator, at step 302, the curatorcompiles personal information of the user and the significantindividuals, such as age, birth date, gender, gender orientation,marital status, name and age of the spouse, number of children, theirnames, and ages, and the interests, activities, religious and culturalaffiliations of every person concerned. At step 303, using the datacollected in steps 301-302, the curator combines the results of profilequestions and stored cross-referenced metadata and compares the resultsto available persona categories. At step 304, the curator determineswhether there is a matching persona category available. If so, thecurator, at step 305, selects and matches the data to select a personacategory for the user from a list of available person categories that ismost similar to the user. Matching techniques used in some embodimentsare described in further details below.

At step 306, if the data indicates that the user's profile issignificantly different from any of the persona categories, thedifference may be quantified and compared to a threshold. If thedifference is sufficiently large, at step 310, the curator may create anew persona category for the user. The new persona category and itscharacteristic user profile may be used for future user classifications.Otherwise, at step 307, the curator selects the closest persona categoryfor the user from the list of persona categories, while noting that theselection is “sub-optimal.” At step 308, based on the selectedsub-optimal persona category, the curator modifies its interactionmodality and frequency with the user. Through further interaction withthe user, the curator may obtain better information to more accuratelymatch the user profile to an available persona category over time (e.g.,step 309, and repeating, as necessary, steps 306-308).

Each persona represents a number of useful user traits. For example, auser trait may be captured, for example, by maintaining an interest log.When certain classes of content (e.g., certain types of images orvideos) are collected or used at frequencies, volumes, or percentagesexceeding pre-determined thresholds, as identified from contentanalysis, social network posts, comments, likes, shares, the user'sprofile may include a trait that represents a specific interest (e.g.the user is deemed to like dogs, take trips, engage in engineering, orparticipate in boating, sports, or certain hobbies).

The following are some examples of personas that may be assigned basedon photographs and videos in the user's media assets:

Persona Indicative Media Assets “New Mom” babies or toddlers, takinginto account the number of individual babies or toddlers, and theirages. “Animal Lover” pets, cats, dogs, rodents, exotic birds, reptiles,or spiders in the media assets; may be further classified based on, forexample, interests in specific breed types (e.g., Yellow Lab, GermanShepard, Dachshund, Poodle, Pit Bull) “Sports Fan” academic, amateur, orprofessional sporting events, types of sport, and team emblems in themedia assets. “Outdoorsman” recreational activities, such as fishingcamping, hunting, hiking, cycling, travel, parties, or gardening.“Photographer” Flowers, landscapes, wildlife “Crafter” handmade items,quilts, pottery “Hobbyist” Redundant Objects (e.g., salt and peppershakers, sports memorabilia, trophies and awards “Activist” Largecrowds, hand-held signs or posters, people in uniform-“SociallyConscious/Recycler” or “Political Activist” based on text depicted onposters “Naturalist” Nature, wildlife, birder, and nature hikes.“Traditionalist” Family- or children-oriented, spiritual, holidaytraditions and celebrations.

FIGS. 4 a, 4 b, 4 c and 4 d shows image Content and contextualindications useful for learning a user profile, according to oneembodiment of the present invention. For example, the curator mayidentify from several images of a child fishing (e.g., one of which maybe FIG. 4 a, showing child 401, fishing rod 402, fish 403, the child'shand 404 holding fishing line 406, and background elements 405 (lake),407 (beach), 408/409 (rocks)) that are taken within a short interval intime that are tagged with a GPS location corresponding to “Keuka Lake,”which is a known recreational scenic location (e.g., waterfront orbeach). The images may relate to an identified significant event (e.g.,“catching a fish”). Techniques for identifying significant events may befound, for example, in U.S. Pat. No. 8,774,528, entitled “METHOD OFSELECTING IMPORTANT DIGITAL IMAGES” to Hibino et al. Salient recurringregions of interest and groups of digital images may be identified, forexample, using techniques disclosed in U.S. Patent ApplicationPublication 2015/0036931, entitled “SYSTEM AND METHOD FOR CREATINGNAVIGABLE VIEWS” by A. Loui et al.

Such images may trigger the curator to initiate a user behavior andcontent analysis. The curator may use additional information which maybe included from other user activities. For example, the curator mayalso learn from the user's multiple social media postings that child 401has been identified as the user's 8-year old daughter; furthermore, thecurator may also learn that the user has shared these images multipletimes with individuals the user identifies as her mother and sister.

As another example, in FIG. 4 b, the curator may recognize person 421holding painting 423 with right hand 422. The curator may predict thatperson 421—who the curator may be able to identify as the user—haspainting as a hobby. Similarly, FIG. 4 d, which shows the user 461holding quilt 462, allows the curator to predict that the user hasquilting as another hobby. FIG. 4 b shows left hand 443 having ring 444on the third finger, with background elements 441 (horizon) 442(lake/ocean). The curator may predict from the image 4c that hand 443belongs to a recently engaged person.

According to one embodiment of the present invention, user behavior andcontent may be analyzed using look-up tables of behaviors and contentthat are previously compiled or dynamically learned. Analysis may beperformed for general categorization or persona identification or forspecific purpose. From the content and metadata, the content may breakdown into categories, such as event types (e.g., family outing, sportingevents), participants, location (e.g., ocean front, lake front, themeparks, camp grounds), general life-style categories (e.g., indoor,outdoor, group activities), and content-type (e.g., skiing, watersports, sailing boats). From the identified participants, the user orother individuals may be classified to demographical groups (e.g.,“mother”, “young child”). Similarly, the location may provide usefulgeographical information (e.g., home zip code). The content breakdownmay provide information relevant to social or network profiles. Thecurator may also take advantage of the user's calendar and contactlists, which allows the curator to predict important events, familymembers, business contacts and friends.

Other indicators of user behavior can also provide information to createuser profiles or personas or to assign individual users to such profilesor personas. Customer interviews, hobbies, relationship types, ethnicbackgrounds, nationalities and national origins, computer familiarity,religious affiliations, political affiliations, preferred news sources,and blogs read by users all provide insights and can be used to assistin establishing a user profile or to assigning a persona to a user.

Many objects and landmarks in images may be automatically identified byimage understanding algorithms, but automatically identifyingindividuals is usually not an easy task. However, there is academicresearch in relationship inference from proximity in photographs.Relationship inference may be achieved taking advantage of similarcharacteristics shared by related individuals, such as facial andphysical features, skin tone, eye color, and hair color. Somerelationships may be inferred based on relative ages (e.g., parent andgrandparent). Certain types of formal photographs may be used toestablish relationships (e.g., wedding photographs in a person'scollection). Such formal photographs have familiar poses, e.g., thebride and the groom are at the center, and parents are likely to bestanding next to the bride and groom. If the best man resembles thegroom, there is a significant likelihood that he is the groom's brother.Parents are likely to resemble either the bride or the groom.

In some context, the garments, apparel or accessories of user, and ofthose with relationships to the user are informative, e.g., formal,casual, uniforms, avant-garde, free spirit, religious garb or head gear,jewelry, tattoos, hair styles, facial hair, climate-related garments(e.g., hats, gloves, coats, or bathing suits). Techniques for derivinginformation from apparels are disclosed, for example, in “ClothingCosegmentation for Recognizing People,” by Andrew C. Gallagher andTsuhan Chen, CVPR 2008, and “Describing Clothing by SemanticAttributes,” by Huizhong Chen, Andrew Gallagher and Bernd Girod, ECCV2012

Additionally, less formal and more casual photographs (e.g., groupsshots used to create holiday greeting cards) also provide valuablerelationship and identity information. Digital versions of these cardsare usually found in a user's collection of photographs and may includetextual information such as, “Merry Christmas from The Hendersons”.Using this example, if such a photograph includes a male adult and afemale adult, accompanied by three children, it is reasonable to inferthe picture depicts Mr. & Mrs. Henderson and their children, who aresiblings of each other. The holiday card may also provide information toinfer the religious or secular background of the holiday season. Therelative ages of the depicted individuals may be determined using agecalculating image understanding algorithms. In other photographs,co-workers, housemates, teammates, classmates, travel companions,relationships, relatives, random connections, or any other grouping maybe inferred from their presence on the social network, from informationor tags entered or the actions taken by users on the social networkingservice. Groupings may vary in granularity (e.g., specified individualconnections in the social networking service, predefined groups of theuser's connections, a particular genre of connections, the user'scollected connections, all connections of the user's connections, or allusers of the social networking service). Techniques applicable for groupanalysis may be found, for example, in “Understanding Images of Groupsof People,” by Andrew Gallagher and Tsuhan Chen CVPR 2009, and in“Seeing People in Social Context: Recognizing People and SocialRelationships,” by G Wang, A Gallagher, J Luo and D Forsyth, ComputerVision—ECCV 2010, pp. 169-182.

The nature and the relative closeness of the relationships can beinferred, although some relationships tend to be temporal, since manyinterpersonal relationships can be transient in nature. This may be truefor friendships in general and is especially true for romanticrelationships. The curator monitors the current status of relationshipsas indicated from social media reports and activities, or through otherinformation sources (e.g., email, text, and telephone interactions).When the curator detects that a user is no longer in a romanticrelationship, the curator will omit the former romantic partner infuture curated content. Other relationship changes can be considered,depending on the natures of the relationships. The curator may followcertain general guidelines, e.g., friendships drift and break and familyrelationships tend to last. The curator may use data mining techniquesto detect whether a friend or a relative is ill or has died by checkingsocial media reports and activities (e.g., Facebook postings). Detectedillness may trigger the curator to recommend a “get well soon” response,and a detected death may trigger the curator to recommend aretrospective on the relationship or a tribute.

Information regarding significant life events may also be collected fromimages. Events may be reoccurring or spontaneous. Some events may beidentified by specific objects present (e.g., hats, decorations, cakes,roast turkey, hamburgers, or pizzas), whether it took place indoors oroutdoors, or the presence of a barbecue grill. A selfie of the user(i.e., a picture taken by the user that includes himself in the picture)just finishing a marathon is probably a significant life event. Inaddition, when the capture device has also recorded contemporaneousactivities (e.g., heart rates recorded by a wearable health or wellnessdevice, such as a FITBIT® device, or pulse rates and blood flow ratesextracted from videos using Eulerian Video Magnification techniques),the metadata may correlate with the picture or video of interest.Collecting such events and metadata over a time period may suggest tothe user significant insights into helping the user maintain a healthylife style (e.g., interactions between the users' weights, heart rates,activity levels, moods, diet or sleep patterns at the times of therelevant life events recorded in the media assets).

The number of individuals present may also provide useful information insome cases. Techniques for inferring relationship and event types aredisclosed, for example, in the article, entitled “Close & closer:discover social relationship from photo collections”, by Wu, publishedin IEEE, ICME 2009, August 2009, pp. 1652-1655. In that article, theauthors disclose using physical proximity in photographs as a surrogateor indication of relationship closeness.

Other events may be inferred based on the people at the event (e.g.,celebrity, famous and infamous individuals, sports figures, actors, teammascots, politicians, and costumed characters). National, cultural, andreligious holidays may be inferred from the user's background andgeographic region. While some holidays are celebrated, others areignored, or discouraged. For instance, Valentine's Day is stronglydiscouraged in India and Iran. The way different cultures and regionscelebrate the same holiday also varies from solemn to festive. Presenceof relatives, close friends, and old acquaintances may suggest family,school, or other social group reunions, for example. A vehicle-relatedevent may be identified, of example, by the counts and mixes of vehicletypes (e.g., luxury cars, antique cars, functional vehicles,recreational vehicles, off-road vehicles, motorcycles, Segways, hoverboards, bicycles, boats, personal water crafts, canoes, kayaks, rowboats, sailboats, wind surfers, aircraft, helicopters, and flyingboats). Similarly, the presence of certain sporting equipment inabundant numbers may be informative (e.g., inline skates and tennisrackets). Concerts and concert types may be inferred by the presence,count, mix and proportions of musical instruments. The number ofperformers and the presence of various instrument types may suggest aband, an orchestra, a choir or a marching band,

Individuals and objects in images and videos that cannot be identifiedwith high confidence by automatic processing (e.g., based on imageunderstanding algorithms) may be manually tagged. The curator may theninquire the user of the tagged items. The queries may be present usinggraphics, in which the curator may be presented as an avatar, or viamore conventional text or audio messaging formats. The curatorpreferably initiates conversational interaction with the user to obviatethe need for learning to navigate or use a GUI or workflow. During itsinteraction with the user, when the curator is unable to adequatelyrespond to a user's request or comment, a remote operator orcrowd-sourced operator may be summoned, so as to intervene and totemporarily take control to resolve the problem at hand. The user neednot be made aware of the involvement of the remote operator, which mayrun in the background, as interaction with the user as the curator maystill be handled by the curator front-end.

In one embodiment, the curator selects a persona type for itself that iscompatible with the demographic persona or user profile determined forthe user. Of course, the selection of the curator's persona ispreferably the user's selection, with options to modify or to select analternative. FIGS. 8 a, 8 b, 8 c and 8 d show different curator personasthat the curator may assume. Some examples are the “Helpful Friend”, the“Professor”, the “Personal Assistant”, and the “Know-it-all Nerd”, shownrespectively in FIGS. 8 a, 8 b, 8 c and 8 d. Depending on which curatorpersona is adopted or selected by the user, the tone of the inquiry canstyled more or less personal, technical or formal. FIGS. 6 a, 6 b, 6 cand 6 d illustrate the tone of the questions that would be asked fromtwo different personas, in according to one embodiment of the presentinvention. For example, the avatar or GUI can inquire, “What is this?”under one curator persona (FIG. 6 a ), or less abruptly “What is thelittle girl is holding?” under another curator persona (FIG. 6 b ).Similarly, the avatar may inquire, “Who is this?” under one curatorpersona (FIG. 6 c ) or less abruptly “Who is this little girl?” under adifferent curator persona (FIG. 6 d ). The user may respond verbally,for example, which may then be converted to text using a voice-to-textalgorithm or application.

In some embodiments, the curator is capable of asking more sophisticatedquestions regarding an image, as more features in the image arerecognized automatically or identified by interaction with the user. Forexample, FIGS. 7 a, 7 b, 7 c and 7 d illustrate the curator persona mayask questions about an image that incorporate already recognized oridentified features in the image, in accordance with one embodiment ofthe present invention. In FIG. 7 a, after learning that one of thefeatures in the image is a “snow fort” (e.g., from a comment associatedwith posting of this image on social media), the curator may ask “Who isthe child in the snow fort?” In FIG. 7 b, the curator exclaims “This isa great shot! What is the dog's name?” In FIG. 7 c, having recognizedthe facial expression of “surprise”, the curator asks, “Who is thesurprised baby with the candy cane?” Likewise, in FIG. 7 d, the curatorasks, “What is the name of the dog wearing the shirt?”

Alternatively, scenes, frames, and video clips containing objects andindividuals to be identified may be forwarded for further analysis at aremote processor where greater image recognition abilities or resourcesexist, thereby augmenting the performance of the automatic recognitionalgorithms executed on the user device. Furthermore, images of objectsand individuals to be identified may be presented for operatoridentification or crowd-sourced assisted identification. The curator mayalso use the remote processor for exchanging information with curatorsfor related individuals (e.g., information that allows birthday-relatedproducts to be offered to a user's siblings or parents).

A number of techniques may be used simultaneously in content analysisfor the curator to determine profiles of the individuals seen in themedia assets and the relationships among them. For example, the curatormay use regression analysis, which is a statistical modeling processthat estimates relationships among variables. Regression analysis mayprovide a relationship between a dependent variable and one or moreindependent variables (referred to as “predictors”). Logical inferencerules, (“If/then rules”) may be employed. For example, when a user isfrequently depicted in images with an identified dog or dogs in general,it may be appropriate to associate the user profile with “dog owner.” Itmay also be appropriate to infer that the user is a “dog person,” ratherthan a “cat person”. In some instances, where the media assets showspecialization (e.g., a preference to a particular breed, such as“Yellow Lab”, “Chihuahua”, “German Shepard”, or “Pit Bull”), the userprofile may be further associated with the specialization. In general,identification of objects in media assets recorded in variousenvironments can be useful. For example, sewing machines, firearms,agricultural equipment, pets, livestock, apartment living, artwork allprovide context and can better assist in identifying the proper userprofile or “persona category”.

The user profile may also take into consideration demographic, contact,and relationship information derived from the user's social networkinteractions (e.g., connections, activities, “checking-ins”, “likes”,“comments”, sharing or postings of videos or photographs). Informationobtained from the user's social network account provides indication astrend data of the user's lifestyle attitudes and preferences at thecurrent time. The user's actual product purchases, user recommendations,and other e-commerce activities are also highly informative. By studyinggeneric purchasing patterns and by monitoring successful andunsuccessful purchase behaviors regarding media-related products (e.g.,photographic products), updating such behavior patterns on a continuousbasis, associations between a behavioral indicator and interest in apotential photo product can be pre-established. For example, the user'saccounts with an online marketplace (e.g., Amazon or Overstock) may bereviewed for the user's styles and activities, based on actual purchasesand purchasing habits.

In addition, motion vector analysis of the video assets may reveal apreference for action (e.g., sports). Alternatively, the user's accountswith streaming vendors (e.g., Hulu, Netflix, or Amazon Prime) can bereviewed to determine the user's viewing interests. Simple movie genrepreferences, such as comedy, romance, sci-fi, westerns, ordocumentaries, and any associated consumer (“star”) ratings maycontribute useful information to the user profile and may also be usedto select and customize suitable media products to the user.

Retail businesses have used personas to represent customers. Creatingsuch personas is a well-established marketing technique. A company oftencreates personas identified to various groups of customer types, thecustomer types characterizing the customers' motivations and obstacles.Such personas help the company understand the customers and help todevelop strategies and techniques to overcome customer reticence withinterventions, incentives, and tailored marketing communications.Personas also allow for different marketing messages to be presented todifferent individuals using the same platform. The frequency and channelof a given marketing message may be tailored to specific personas, andto enable accurately predicting sales based on prior sales. Internetadvertising companies rely on personas developed from monitoring users'browsing history, social network activities, and any advertising or“pop-up ads” the users select or responded to or clicked on. Informationderived from personas allows marketers to advertise their merchandise toa targeted and potentially interested audience.

Personas may be created and assigned with a hierarchical ontology. Asdata is collected from the user, the user's media collection and theuser's media activities, a more specific persona is assigned or created.The persona type may have different levels with different granularities.For example, at level 1, a user may be assigned “Sports Enthusiast”persona based on the user's taking photographs at sporting events. Theuser may also be assigned, at level 2, a “Sports Mom” persona, when itis recognized that the user takes significant number of photographs ofsporting events in which her children participate. The user may also beassigned, at level 3, a “Gymnast Mom”, when it is recognized that asignificant number of photographs are taken of the user's childparticipating in gymnastics.

Additional discussion regarding personas may be found, for example, in(a) the book “Psychological Types. Collected Works of C.G. Jung” byJung, Carl Gustav (Aug. 1, 1971), published by the Princeton UniversityPress (ISBN 0-691-09770-4); (b) the article “The Power of the Persona,”by Rind, Bonnie, May 5, 2009; (c) “Persona Management,” by Bob Pike,published in Computer Fraud & Security 2010 (11): 11-15.doi:10.1016/S1361-3723(10)70145-7; (d) the article “The origin ofpersonas,” by Alan Copper, Cooper Journal, May 15, 2008; and (e)“Getting from research to personas: harnessing the power of data,” byKim Goodwin. Cooper Journal, May 15, 2008.

Creating a series of personas for users allows a company to group andcategorize potential customers' and actual customers' purchasingbehavior and activities. Personas are surrogates for in-depth userprofiles and provide a company an ability to tailor the marketingmessage and to produce product offerings to selected groups of potentialconsumers. Personas are also useful if only incomplete profileinformation is available for particular users and new users. Personascan provide information on likely or preferred product categories, suchas photo calendars, posters, photo albums, or digital slide shows. Thisinformation allows a company to appropriately scale up and stock thematerials, media, and products based on projected sales to the variouscustomer groups represented by their corresponding personas and tocustomize individual products with digital multimedia assets from theindividual user's own multimedia assets or accessible assets from otherusers.

These personas can also be used to aggregate user behaviors and to makeanonymous any individual user's actual behavior. The resultinginformation can be used to plan for manufacturing and distribution. Inaddition, this information can be provided to other advertisers andmanufacturers. The accumulated buying, social network interaction,viewing and listening histories may be used to determine or affect a“persona” model, which serves as a surrogate for the user or a new usergroup. Once categorized, the personal data of a user need not be kept bythe company and can stay resident with the user. Important dates andrelationships and access to all image accounts may be used, with userpermissions, so that potential photographic products can be offered in atimely manner. The number of personas available and the personacategories available for assignment to a user can be modified based onproduct sales, user feedback, and other factors. Special promotions andoffers can be provided to different persona groups.

As mentioned above, the user profile may be actively created by usersubmission (e.g., by filling out a form), an interview with a human or adigital avatar, or a series of interactive text or audio questions. Ofcourse, personas can be created in advance or created dynamically fromobserving behaviors of the participating users, or modified to reflectchanging user behaviors. A user persona is a surrogate for the userprofile and allows the product or service provider to gauge and estimatethe shares of various user types in its user base. In addition, activelymonitoring the effectiveness of persona classifications with targetusers can provide quantitative results into what works for your targetusers and what doesn't. Other techniques for developing personasinclude, for example, sentiment analysis or “opinion mining”, which is atechnique for assessing consumers' emotional states. Sentiment analysisprovides context for text provided by users from comments, blogs,reviews, and posts on social networks. Sentiment analysis processes textinformation to determine a sentiment, theme, interest, concern, or othersubjective information regarding objects such as a topic, people ofinterest, brands, or companies.

According to one embodiment of the present invention, each user isassigned to one or more personas. To preserve anonymity when usingservices from other application programs, only the curator monitors theuser's account activities, the user's identity, account information andother personal or confidential information are made anonymous beforeproviding them to other application programs, such as content analysisor social network activity analysis. The curator retains control of theuser's identity information and individual photographic product purchasehistory, for example. A single individual may be associated withmultiple personas using persona hierarchies or a system of relatedpersonas. For example, an individual who is both a young mother and agrammar school teacher may be categorized by separate personas todifferentiate the relationships and events of her personal life and theevents and relationships of her professional ones (e.g., school-relatedevents would be presented separately from personal events). Thecurator's text messages or pop-up ads related to these personas, whileappearing on the same user's device, would be differentiated (e.g.,addressing the user differently). For example, the curator would addressthe teacher or professional persona as “Mrs. Johnson”, while the youngmother persona would be addressed as “Betty”. Personas can be purchased,acquired, or shared on social networks, search engines, or shoppingservice providers.

In addition to predetermined user profiles or personas, the system maydynamically recognize and assign users to profiles that are learned overtime. In one embodiment, the method of frequent itemset miningidentifies common characteristics for sets of users. For example, thesystem may recognize as a class of users females age 20 to 35 whofrequently photograph infants. While this set of users is similar to the“New Mom” persona identified previously, this set of users wasdynamically learned by the system, without an express association of apre-defined label (e.g., “New Mom”) with the persona, and withoutparticipation by a human operator or system designer to define thepersona. When a sufficient number of users fit such a dynamicallydiscovered profile, the system may monitor the behavior of the users, soas to learn more and more characteristics that can be associated withthe persona.

In an alternative embodiment, the system uses a simplified form offrequent itemset mining by which a predetermined class of categories isused to dynamically build personas. In particular, rather than usingfrequent itemset mining to fully discover the possible characteristicsof a persona, specified characteristics such as user gender, age andlocation are combined with, e.g., image tags, to form a more limited setof possible personas.

The curator may detect a “traveler persona” in a user based on a GPSanalysis of a person images. Likewise, the curator may detect thatanother user is a “non-traveler” based on ground truth GPS informationthat indicates that the user does not venture far from home on a dailyor weekly basis. The user's non-traveler status may be confirmed, if theuser takes sufficient pictures to ascertain by GPS analysis his or hernon-traveler behavior. Based on these groupings, a traveler persona maybe defined to represent an individual who ventures more than a fewhundred miles from home for days on end and takes a number of picturesover a predetermined time period beyond a settable threshold. AnalyzingGPS patterns for travel frequency, trip distance, destinations, andphotographic activities may be used (a) to enhance the user's profile,(b) assign an appropriate persona to a user, and (c) modify or createnew personas. The suggested products may include personalized travelmugs or backpacks, personalized luggage, bragging products likepersonalized postcards from locations the person has traveled to (e.g.,Italy, Paris, Grand Canyon). By analyzing GPS locations and comparingthem to a known set of vacation locations, certain cities, theme parks,typical vacation spots, the curator may ascertain a persona type. Thecurator may mine the GPS data and suggest locations that the user hasnot yet visited and suggest places to go, to take pictures and to makeproducts.

The set of defined personas may be linear or hierarchical. For example,the system may have a persona corresponding to “sports moms”, where thatpersona may have been manually or dynamically discovered. The class ofsports moms may be further specified into for example, “T-ball moms,”and “soccer moms.” If the system is then presented with images ofchildren engaged in another new activity, such as gymnastics, wheregymnastics is ontologically recognized to be a type of sport, then thesystem may automatically associate general characteristics of “sportsmoms” to the newly discovered class of “gymnastics mom”. This methodallows attributes to be associated with a persona even with limitedsamples. As additional data is obtained, the system may refine a personadefinition. In some embodiments, selected characteristics of ageneralized persona may be overridden to define a specialized persona.Information may flow up or down a hierarchy. For example, a system mayinitially have defined personas for “T-ball moms” and “soccer moms.” Outof these personas, the system may form a generalized persona “sportsmom”, which can then form the basis for a newly discoveredspecialization “gymnastics mom.”

In addition, the curator is sensitive enough to specific cultural orethnic preferences and biases to avoid including an individual in apersona category that would inconvenience or displease the user. Thepersona may evolve over time, based on additional purchases made by theuser or other subsequent data. Persona may be further personalized basedon the user's behavior, to capitalize on trends and even fads (e.g.,selfies and planking). A persona change or personalization may betriggered, for example, by recent content showing specific behavior bythe user (e.g., planking). Over time, persona assignments may beevaluated for persona effectiveness (e.g., correlating user profile andmetadata analysis with media asset sharing and sales). The effectivenessof the curator's persona may also be evaluated (e.g., proactive, subtle,or conversational approaches correlated with content and profile).

Tags derived from characteristics of a group of users' media collectionscan be used for generating a persona for the group. Under this approach,a persona profile can include the distribution of certain tagsassociated with the users' media collections. For example, a group ofyoung mothers may have a tag profile {baby, child, girl, boy, play, joy,indoors, family, portrait, cute, smile, fun} derived from their imagecollections and associated with their persona profiles. Likewise, apersona profile for a group of cycling enthusiasts can have the tagprofile {bike, trail, outdoors, morning, exercise, road, people, action,leisure}. The tags in the profile may also have a score based on theimportance of the tag in the profile. For example, the tag ‘indoors’ inthe tag profile may have a low score because it is not necessary forimages to be indoors for that persona profile; whereas the tag ‘family’may have a high score in this group.

Appropriate demographic categories to characterize a persona profile mayinclude: age, gender, occupation (e.g., “homemaker,” “part-time worker,”“hourly worker,” “skilled worker”, “trade artisan,” “professional” or“executive”), income level (e.g., salaried or household income levels),location (e.g., “urban,” “suburban,” “rural” or a ZIP code), educationlevel (e.g., “High School,” “Technical School,” “Some College,”“4-YearDegree,” “Master's Degree,” “Professional Degree,” or “PhD or otherdoctorates.”), and relationship status (e.g., “married,” “single,”“divorced,” or “alternative”).

The curator may, in addition to an initial interview, occasionallyfurther interview a user to refine a user profile, assign an up-to-datepersona to the user, or to dynamically modify the persona to reflect achange in the user's attitudes or in response to newly identifiedopportunities.

In one embodiment, the tags derived from users' collections of mediaassets may be used for persona profile generation. A persona profile maybe viewed as a distribution of tags associated with the persona. Forexample, a group of young mothers may have a tag profile {baby, child,girl, boy, play, joy, indoors, family, portrait, cute, smile, fun}derived from their image collections and associated with the personaprofile. A persona profile for a group of cycling enthusiasts may havethe tag profile {bike, trail, outdoors, morning, exercise, road, people,action, leisure}. Tags may be assigned a score in the context of the tagprofile based on the perceived importance of the tag in the personaprofile. For example, the tag ‘indoors’ may be assigned a low score inthe cyclists' persona profile because “indoors” may play little part forimages for that persona profile. Conversely, tags in the associated tagprofile are assigned high scores. In some instances, certain tags may benegatively associated with a particular persona; i.e., the presence ofsuch a tag is a negative predictor for the persona.

Persona profiles can be generated using a frequent itemset-miningapproach, described below, on a very large set of users to gather tagsthat commonly occur together. A group of frequent itemsets that aresignificantly different from each other can be chosen as a set ofpersona profile to describe the users. A user may be assigned to one ormore personas based on matching the tags of the user's collections ofmedia assets with the tag profile of each persona. A weight may beassigned to the user for membership in each assigned persona, so as toindicate the degree to which the tag profile matches.

Another technique for developing personas is progressive profiling whichuses multiple choice questions, forms, and directed questioning to usersto collect customer insights and opinions, which accumulate and becomemore detailed over time. These interviews obtain information directlyfrom the customers and prospects. Regional sales teams that engage withcustomers and retailers can also be interviewed for observations andinsights about the customers they serve and their preferences. Inpractice, the curator can carry out progressive profiling using text oraudio interactive interviews.

According to one embodiment of the present invention, the accumulatedbuying, viewing, social network interactions, listening history, tagsand demographics of users are stored using any suitable transactionaldatabase format, such as that described in the book “Data MiningConcepts and Techniques,” by Han et al., published by Morgan KaufmannPublishers, Boston, pp. 14-15, 2006. Under this approach, a“transaction” with a unique transaction identifier (“UserID”) isassigned to each user. Each transaction is of the form of a tupleconsisting of a UserID and a set of quantized descriptors (i.e.,(UserID, Descriptor1, Descriptor2, . . . , DescriptorN)), where thenumber of quantized descriptors may be different for each user. Someexamples of quantized descriptors may be “mom”, “28-32 yrs”, “MediumIncome”, “Teacher”, “Honda CRV”, or “Environmentally Conscious”. Thequantized descriptors for each transaction represent a set of “items”which, including any of its subsets, are collectively referred to as an“itemset”.

A frequent pattern-mining step is carried out from time to time toidentify recurring patterns that are present in the transactionaldatabase. The frequent pattern-mining step identifies frequent itemsets.The frequent itemsets are co-occurring descriptor groups that occur inat least a predefined fraction of the users. The frequent pattern-miningstep may use any suitable method known in the art to identify thefrequent itemsets.

In one embodiment, the frequent itemsets are determined using thefollowing method, which uses Γ to denote the set of all possiblesymbolic descriptors in the transactional database, F⊆Γ to denote an“itemset,” and transaction τ_(i) denotes a transaction having avariable-length itemset associated with the i^(th) user in thetransactional database. Then τ=<τ₁, . . . , τ_(i), . . . , τ_(n)>denotes the transactional database, with each transaction including anitemset corresponding to the corresponding user.

Therefore, for any itemset, F:

cover(F)={τ∈τ|F⊆τ}  (1)

That is, cover(F) denotes the set of transactions τ in the transactionaldatabase τ containing the itemset F, and therefore are to be counted inthe frequency of F. Let support(F) denote the size of cover(F) (i.e.,the number of transactions in cover(F)):

support(F)=|cover(F)|  (2)

where |A| denotes the number of elements in the set A.

Frequent itemsets Φ denotes the set of itemsets having support(F) atleast “minsup” (which is a predetermined minimum size):

Φ={F|support(F)≥minsup}  (3)

In one implementation, “minsup” is a value in the interval [0,1],representing a fraction of the total number of transactions in thetransactional database. For example, in one implementation, if a“persona” category must have at least 2% of all users, “minsup” may beset to 0.02.

Using the above method, the following persona was obtained in oneimplementation:

-   -   “Betty”—new mom, 28-32, medium income, teacher, Honda CRV,        environmentally conscious    -   “Belinda”—new grandmother, 55-60, fixed income, retired, Ford        Focus, family matriarch    -   “Bob”—2^(nd) marriage, 35-38, medium income, retail manager,        Jeep Cherokee, sports fan    -   Veronica—“Creative Crafter”    -   Judy—“Power Mom” 32-42, 3 Children (grammar school)    -   Bill—“Hipster”    -   Tom—“Action & Adventure”    -   Sally—“Proud Grandma”    -   Jeanne—“The Collector”    -   Cindy—“Family Organizer”    -   Dan—“The Hobbyist” 50-56, 3 children, 5 grandchildren

An algorithm for frequent itemset pattern-mining in a transactionaldatabase may be, for example, the Eclat algorithm disclosed in thearticle “Efficient Implementations of Apriori and Eclat” by Borgelt, inthe Proc. of IEEE ICDM Workshop on Frequent Itemset MiningImplementations, 2003.

Furthermore, usage statistics from a large body of users may be obtainedby performing frequent itemset pattern-mining within each specificproduct (e.g., “Mother's Day card with flowers theme”, “Photobook withDisney World travel theme”, or “5-image collage with family fun theme”).The itemset pattern-mining may lead to identification of characteristicsgroups that relate to that specific product. The product could then berecommended to other users with similar characteristics.

Personas are a particularly useful tool for creating customized mediaproducts or photographic products for the users because they allow acompany:

-   -   (a) to understand the appropriate/intended emotional response        for large demographic populations;    -   (b) to build products to satisfy the large demographic        populations;    -   (c) to target-market only those customers most likely to        purchase, and to avoid making offers that may alienate potential        customers;    -   (d) to identify important or special content to trigger a        spontaneous selling opportunity;    -   (e) to tailor the product offering to the intended customer; and    -   (f) to be able to predict sales to the large demographic        populations rather than individuals for efficient production,        predictable sales, and inventory control.

Generally, customized media products may be physical or virtual. Therequired resources for virtual products, including computational andnetwork capacities and bandwidth, are generally predicted and additionalresources can be incrementally deployed. Such additional resources maybe provisioned from third party vendors during times of peak demand.

There are at least two classes of physical customized media products:(a) generic products that can be customized for individual customers and(b) event-specific products that are designed to be customizable or canbe included with customized products. A certain level of inventory isrequired to meet demand. Examples of inventory for generic productsinclude rolls and sheets of paper for creating photobooks, posters, orcalendars, printing ribbons, inks, toner, pre-scored greeting cards,blank mugs, t-shirts, apparel items, decor items, mouse pads, albumcovers, and the like. Generic products can also be customized andproduced by third party vendors who are provided with appropriatecontent and specifications.

While the inventory for creating generic products may not expire,event-specific products are specific to a time period or event, so thatthe inventory required may diminish in value or render valueless afterthe specific time period or after occurrence of the specific event. Forevent-specific products, an accurate estimate of the demand is importantto plan the required inventory level. Excess inventory would need to bedisposed of, unless the event is recurring. For recurring events, theinventory can be stored until the next occurrence of the event, therebyincurring additional cost and may even incur losses.

Some examples of specific-event products are: a World Cup soccer-themedmug or soccer ball on which a user can insert personal pictures, aValentine's Day-themed heart-shaped card with pocket to insert candy, aphoto t-shirt with a political slogan for supporting a candidate foroffice, a photobook with motifs celebrating the 100^(th) anniversary ofa theme park, photo frames with NASCAR drivers and cars, etc. In each ofthese cases, there is either a specific date or time period after whichthe inventory has little value. Alternatively, as with the Valentine'sDay-themed heart-shaped card, the inventory would have to be stored fora year until the next Valentine's Day.

In addition to generic and event-specific products that requirecustomization, “companion” products can be inventoried for accompanyingthe main media product in a gift bundle. For example, a generic mugcustomized by an image of the recipient's new grandson may beaccompanied by a choice of an assortment of teas, coffees, or hotchocolates. The concept of the value in inventory can be generalized tothe period of time between event recurrences. Single-event items, suchas an item specific to New Year's Eve 2017, have little value after thatevent. Holidays and seasonal items have value annually, but incurinventory and storage costs. A calendar service that offers just-in-timegenerated calendar pages may be seen as having a monthly period of use.Other generic items are effectively continuously valuable.

When including a customizable product in the product offerings, a set ofuser personas may be associated with the product to indicate who thetarget customers are. A probability score may be computed to indicatethe likelihood that a customer associated with the user persona may buythe product. The probability can be computed based on, for example, thecustomer's disposable income, a level of interest in the event theproduct is associated with, the number of the customer's social networkcontacts who are also interested in the event (thus, who are potentialrecipients of a gift of the product), the customer's buying history, andthe customer's demographic profile. Based on this probability, anestimate of the quantity of products can then be generated based on thenumber of customers who are associated with the persona and the averageprobability of purchase. A small buffer can then be added to ensurethere is sufficient inventory for the product.

For events that are recurring (e.g., Valentine's Day and Christmas),previous years' sales data may be used to calculate a requiredinventory. Using the historical data as baseline, adjustments to therequired inventory can be computed based on changes in the populationsof customers associated with different personas.

For virtual products (e.g. slideshows, video summaries and highlights,and mini-movies), the television may be a better display medium than thedisplays on mobile devices. Even for physical products (e.g.,photobooks, calendars and collages), a preview on a large screenprovides a better viewing experience by allowing the user to select thephysical products from their virtual renditions. In addition, the mobilephone is a utilitarian device that many users associate with tasks andwork, while the television is a device that the users associate withtheir leisure hours, when they can be entertained by their images andcan engage with the curator. The curator of the present invention may bepart of a system that includes a streaming channel (e.g., the “KODAK®Moments” channel) that is accessible using streaming devices (e.g.,ROKU®, BLU-RAY™ players, APPLE TV®, and GOOGLE CHROMECAST®). When a useraccesses the channel, the user is presented choices of recommendedproducts that the curator creates for the user. The user may select theproducts to view. Previously presented products that have not beendeleted by the user remain on the channel for re-viewing.

The curator takes into consideration the required resources (e.g.,computational and network capacities and bandwidth) when recommending avirtual product. When a user is reaching a resource constraint, e.g.,the user has only a low bandwidth connection, the curator does notsuggest at that time products that will require a significant amount ofsuch resources to fulfill or preview. Instead, the curator waits tosuggest such products until the user has greater access to resources(e.g., a WiFi network connection), or their resource allocation isimproved.

Content evaluation may also be performed for a specific purpose; in oneimplementation, content evaluation is performed for automatic “virtualproduct creation” relevant to a particular occasion or “trigger event.”A trigger event may be an upcoming planned or spontaneous, generic orpersonal event to be recognized, celebrated, shared, or memorialized.Typically event triggers are specific dates or timeframes identifiedfrom network-connected users' calendars, social network accounts, posts,comments, and the likes, and may include birthdays, anniversaries,graduations, new jobs, new homes, births, deaths, weddings, nationalholidays, cultural holidays, and ethnic holidays, special regionalevents (festivals, art shows or fairs) or seasonal events (e.g., winetours, vacations or races). For example, the curator may notice thatseveral pictures were taken in quick succession of the user in campclothing, outdoors on the shore of Keuka Lake, holding a very largefish. This event triggers an opportunity for the curator to express, forexample, “Wow, that's quite a catch! Would you like it posted on yourFacebook Timeline? A framed picture would look great . . . ”

The curator can also initiate questions during picture taking sessions,such as “What is that red object that you just took several picturesof?” The user may choose to ignore the question or to respond. Thecurator may then convert a verbal response to an image tag (e.g., “It'san antique fire hydrant.”). The curator would consult its library ofterms for “antique fire hydrant.”

Event triggers can be adapted to accommodate different regional,cultural traditions, customs, demographics, ethnicities, and otherpractices. Some techniques for content evaluation based on eventtriggers may be found, for example, in U.S. Pat. No. 9,037,569, entitled“IDENTIFYING PARTICULAR IMAGES FROM A COLLECTION,” to Wood et al., whichdiscloses indexing individual images based upon their capture date andtime, and mapping the date and time to concepts such as seasons of theyear or other temporal periods. In addition, the '569 patent teachesassociating information in personal calendars to the media assets basedon civil and religious holidays, birthdays, anniversaries, or specificpersonal events (e.g., “Florida vacation”). As another example, U.S.Pat. No. 8,634,662, entitled “DETECTING RECURRING EVENTS IN CONSUMERIMAGE COLLECTIONS” to Das et al. discloses other applicable techniques.Similarly, music tracks may be collected automatically according tothemes or keywords using techniques that are found, for example, U.S.Pat. No. 9,098,579, entitled “AUTOMATICALLY SELECTING THEMATICALLYREPRESENTATIVE MUSIC” to Luo et al.

Content evaluations can also be triggered by “content triggers.” Acontent trigger may be a special occurrence recorded and identified inthe content or metadata, which may be related to a time or date, alocation, an event, or an activity. Content triggers can be adapted toaccommodate different regional, cultural traditions, gestures, customs,demographics, ethnic, and other practices. The curator may takeadvantage of content or event triggers to offer or promote a product orservice. In addition, the curator may engage the user to acquireadditional information about the image content and context examined, soas to better understand the importance and relevance of the image,objects, and individuals analyzed.

The following content triggers may be identified in the media assets tobe examined:

Content Trigger Type Triggers Relevance Interesting location orLocations: theme parks, May be hierarchically seasons sports stadiums,recreational adapted, relevant to venues, trails, beaches, personadesignation ocean, lakes, waterfronts, mountains, restaurants, museums,coffee shops. Seasonal activities: Spring or summer: water sports,boating, sailing, skiing, hiking, mountain biking, hiking, climbing.Fall or winter: hiking, skiing, snow-shoeing, sledding. Presentingpresenting an object or Special event (“I’m objects: a fish, deer orother Special”): the objects games, a ribbon, a medal, a being presentprovides trophy, an award, artwork, a context and significance craft, apet, flowers, an (e.g., birthday, cultural animal, a birthday cake, aevent, academic or sport gift or present, formal attire. achievements)Posing Posing in front of natural Special Events (“I’m and artificial,scenic Here”): travel, vacation, landscapes, landmarks, field tripsobjects or structures. Groups Individual or group; object Work or socialevents, in foreground or objects present may background. provide contextof event Gestures General expressiveness: Emotion or mood, may holding,pointing, palms up, have specific or different palms down, opened fist,meanings, positive or closed fist, hands folded, negative, in differentthumbs up, thumbs down. cultures. For example, Unusual expressiveness:in western cultures, eye gaze or rolling, “duck nodding downward face”,winking, making acknowledges an faces, sticking a tongue out, unfamiliarindividual, diving, jumping, planking (a while nodding upward cultic,lying-down game), nod acknowledges a posing in Hadoukening or familiarindividual. Makankosappo stances, Gestures indicate that batmanning,etc. the subject or subjects in the image performed something special orotherwise made the image noteworthy. Dynamic gestures, poses, videosequences with time Same as in still images, and actions. or locationreferences. only richer content. Proxemics (i.e., the study of Proximityamong Significance depends on the nature, degree, and individualsappearing in an social and interpersonal effect of the spatial imagesituations, and separation individuals environmental and naturallymaintain) cultural factors. Individual proximity in photos may indicateimportance.

According to one embodiment of the present invention, a photograph thathas been analyzed includes the following metadata:

Objects Identified: “fishing pole”, “mountain”, “shoreline”, “lake”,“fish”, “female child”, “ Betty Johnson”, “boat”, “left hand -presenting”, “expression - smile”, etc....  time: 11:32 EST  date: 25JUL 2015 SAT  location: LAT: 42.585876 LONG: −77.082526   locationidentity: address: 555 Eastlake Rd. Penn Yan, NY 14527   details:“Johnson Family Cottage”   environment: seasonal, recreational,waterfront, beach, cottage,   boating, fishing  trigger condition: 1  trigger type: fishing, catching fish, presenting fish  number ofindividuals: 1  identity: Betty Johnson  relationship: daughter birthday: 02 JUN 2004  wedding date: 0  children: 0  detected language:American English  confidence: true ( value : 1 )  pose: roll (−0.33),yaw (1.25) ,pitch (−2.31)  race:  Caucasian (0.92)  face brightness :0.62  face sharpness : 1.4  emotion : calm: 73%, happy: 45%  age : 11.3(value : 11.3)  smile: true (value : 0.92)  glasses: no glass (value :0)  sunglasses: false (value : 0)  hat: false (value: 0)  beard: false(value : 0)  mustache: false (value : 0)  eye_closed: open value: 0) mouth_open_wide: 0% (value: 0)  beauty: 96.22 (value: 0.9622)  gender:female (value: 0)  zoom ratio: (eye separation distance/frame size) share to Twitter: false (value : 0)  share to Instagram: true (value: 1)  share to Facebook: true (value : 1)   likes: 27   shares: 11  comments: 3    “What a catch!”     “Fish fry tonight Betty.”    “YourGrand Dad Jake would be so proud of you Betty.”

After indexing the media assets or, if the media assets have alreadybeen indexed (e.g., metadata available from a 3^(rd) party service), thecurator organizes the indexed metadata relative to the user's profileinformation. Some images may be identified as “special images”, whichmay be set aside as potential assets for an image product or gift, as ameans to associate the appropriate persona with the user, to discovernew imaging fads, trends, and behaviors, and to understand the user'simage taking or usage priorities (e.g., documenting, celebrating,sharing, bragging, gifting, or memorializing).

Based on the results of analysis, the curator may recommend products tothe user. Techniques for image product recommendation are disclosed, forexample, in U.S. patent application Ser. No. 62/273,641, entitled“METHOD FOR IMAGE PRODUCT RECOMMENDATION”, attorney docket no.041667-420011. The curator may identify “important images”, based onesthetic rules, image usage and capture patterns, social interests, andrelationships with other individuals depicted in the images orassociated with them, for potential use in image products. For example,the analysis may infer “sentimentality” in a user who has recordednumerous images of cats in a home location indication. It may furtherinfer that the user is likely to enjoy cat-related graphics oncustomized media products. The curator may make suggestions andrecommendations that are intended to educate the user about the possiblesharing, creative, and gifting opportunities presented by the user'sphotographic and video collections.

The recommended products may include, for example, crowd-sourcedcreation of virtual products and promotional virtual versions ofpotential hard copy products. The operators may be paid employees of thecurator's creator or private contractors in a competitive environment(e.g., the artists of the “ETSY” market). Some services, such as the“Easy as Amazon” service allows creation of customized photographicproducts and photographic product types that includes the user's orothers' images to be created and offered to the user. The curator mayalso receive input information to further customize a product forspecific individual or occasion. FIGS. 9 a and 9 b show two forms 580and 960 that the curator may present to a user to customize a greetingcard to be created, in accordance with one embodiment of the presentinvention. As shown in FIG. 9 a, the curator receives a profileregarding the recipient of the greeting card 590 (“Card for . . . ”).Similarly, in FIG. 9 b, the curator receives a profile of the sender970. Note that, in this instance, the curator is made aware of culturaland ethnic sensitivity in creating the image products it recommends. Forexample, FIGS. 9 a and 9 b show that the recipient is a male KoreanBuddhist whose primary language is Korean (denoted by items 600, 610,690, 700, 730, 740, 890, and 900), while the sender is an American malewho religiously agnostic and speaks American English (denoted by items980, 990, 1070, 1080, 1110, 1120, 1150, and 1160). Information about thesender or recipient may be provided via buttons on the GUI (620, 630,640, 670, 680, 710, 720, 750, 760, 790, 800, 830, 840, 870, 880, 910,920, 930, 940, 950, 1000, 1010, 1020, 1050, 1060, 1090, 1100, 1130,1140, 1170, 1180, 1190, 1200, and 1210). Other types of informationabout the recipient and sender may be provided too (e.g., age (650, 660,1030, 1040), relationship (770, 780), emotional intent (810, 820), andoccasion/event (850, 860)). Techniques for such products that can beused may be found, for example, U.S. patent application Ser. No.62/255,239 entitled “CROSS CULTURAL GREETING SYSTEM,” attorney docketno. 41667-419011. Other image product creation techniques may be found,for example, in U.S. patent application Ser. No. 14/629,075, entitled “AMETHOD FOR OUTPUT CREATION BASED ON VIDEO CONTENT CHARACTERISTICS,”filed Feb. 23, 2015.

In some embodiments, the curator extends beyond serving the initial userto the recipient or recipients. For example, the curator may suggestthat the user create a framed photograph gift product for a recipient(e.g., the user creating a framed picture of her daughter as a gift tothe user's own mother). In this instance, the curator also recommendsthat the user invites the grandmother to participate in completing thefinal details of the gift. Such final details may include, for example,picking the frame color and style from a collection of different colorand different style frames. The curator may engage the recipient (i.e.,the grandmother) to present her possible product previews and theallowable customizable options within a range of costs and options theuser has authorized. (The curator may offer the recipient furthercustomization to the product at the recipient's own expense.) Thecurator may also have gathered enough information about the recipient tohave associated the recipient with a persona. In that case, based on thegrandmother's persona, the curator recommends to the grandmother aparticular frame style (e.g., Victorian), or a particular frame color.The curator may exploit knowledge about the grandmother (e.g., thegrandmother's house décor) to suggest that a particular frame colorwould accent the colors of her living room. The final product variationas decided by the recipient would be delivered to the recipient, with anoptional notification sent to the user of the final choice.

When evaluating an event trigger or recommending a product, the curatortakes into consideration the motivations and obstacles todecision-making by the user. Some typical motivation categories that canbe incorporated into the assigned persona are: (a) celebrate orrecognize important life events, (b) connect or share with friends andfamily, (c) sharing creativity, (d) recording and sharing hobbies,crafts, collections, or (e) sharing the excitement or bragging rights.Obstacles to decision-making may include: (a) “I'm too Busy”, (b) “I'mnot Creative”, (c) “It's too Hard”, (d) “It's too Expensive”, and (e)“I'm concerned about Security or Privacy.” Based on the motivations andobstacles assigned to the persona, the curator may suitably intervene,for example, by alerting the user prior to an upcoming event, takinginto account (a) the time required to produce and deliver customizedproduct, and (b) the time window for user requires for his or herdecision. Based on these factors, the curator may present the virtualproduct on a preferred day and time appropriate to the persona. Asuccessful intervention (i.e., one that results in a purchase decisionby the user) requires the curator to intervene with a timelypresentation, or perhaps also offering an incentive (e.g., free shippingif the user orders within a specific future time window).

In some embodiments, when evaluating an event trigger or recommending aproduct, the curator further considers the current environmentalcharacteristics of the user. Environmental characteristics may includethe ability of the system to generate and deliver either a productpreview or the actual product to the user. For example, if the user iscurrently connected to the Internet via a low bandwidth data connection(e.g., a 3G cellular network), or the user is close to approaching adata cap on their cellular plan, then the curator will not recommend anyproduct that requires a large amount of data be transferred between theuser's device and the system to display. When the user's environmentchanges, for example, after the user connects to a Wi-Fi network, suchomitted products may then be presented. Similarly, if the user has botha smartphone with a small display and a tablet device, the curator willnot recommend a product that requires a larger display to be adequatelypreviewed when the user is using the smartphone. The curator may waituntil the user is interacting with a more appropriate device. In someembodiments, the system may also consider the user's current activity ona device. A complex product such as a photobook would not be shown whenthe user is expected to be only briefly interacting with the device.Instead, such a product would be displayed at a time when the systempredicts the user to have a greater degree of leisure or freedom to forthe product preview.

In one embodiment, a tool for creation of a media-centric productincludes look-up tables. FIG. 10 a shows a look-up table 1220 that canbe used by the curator to a profile of the desired product, according toone embodiment of the present invention. As shown in FIG. 10 a, thecurator selects a product based on five profile categories: relationship(1230), emotional intent (1240), occasion (1250), cultural profile(1260) and religious affiliation (1270). Each entry under a profilecategory may be associated with a characteristic of the media-centricproduct to be created. FIG. 10 b shows components of a greeting cardthat can be selected based on the values assigned to one or more of theprofile categories. For example, if “romantic relationship” is selectedunder relationship 1230, “romance” is selected under emotional intent1240, and “Anniversary” is selected under occasion 1250, the curatorwill be directed by the look-up table 1280 (FIG. 10 b ) to cardcomponents “red-roses” under “flowers” component 1310, and “respectful”under graphics 1300, along with, optionally, appropriate font colors1290, symbols 1320, and language 1330.

In presenting the virtual product, the curator may use marketingtechniques, such as:

-   -   (a) content, relationship, or event-related pitches;    -   (b) randomly selected products, but logged to prevent repetition        in future pitches; and    -   (c) personalizing the approach with the inclusion of specific        names, relationships, and events; for example:        -   “Share your Creativity with ______,”        -   “Recognize the Accomplishment of friends and family.”        -   “Celebrate with the ones you love.”        -   “Make ______ feel special.”        -   “Treat yourself.”        -   “Reconnect with family and friends.”        -   “Let them know you care.”        -   “Make ______'s day.”        -   “Share the good times.”        -   “Cherished Memories.”        -   “Celebrate ______'s new job.”

The following are two examples that illustrate the operation of thecurator:

Example 1: Early in December, a user posts on her Facebook account aseries of photos her new baby, Audrey. Her Netflix account indicatesthat she has streamed the title “Yule Log Fireplace” and severalholiday-themed and romantic comedy movies. Her Amazon account indicatesalso that she has recently purchased Christmas decorations. Based onthese user behaviors, media selections, the purchase dollar amount andfrequency, the curator presents several virtual customized photographicproducts for the user's purchase that features Audrey's image. Thespecific products are selected based on the Amazon purchase history. Forexample, if the user's purchase on Amazon exceeds a predeterminedthreshold dollar amount, the user will be first offered a three-pieceornament set each inset with a different picture of Audrey. However, ifthe user purchase was less than the threshold amount, the user will beoffered a single ornament with Audrey's picture. Based on the styleextracted from her Netflix purchases, the ornaments offered to the userwill be in the “Cute or Fun” or “Traditional” categories. If the offeris successful (i.e., the user purchase the offered ornament orornaments), the sale is noted and will be taken into account in a futureupdate, according to behavior-based product selection algorithmimplemented in the curator.

Example 2: Early in December, an individual identified as the user'smother (“Grandma”) commented on a series of photographs of hergrand-daughter, Audrey, posted by the user on her Facebook account.Grandma shares Audrey's pictures with her friends and family, taggingthe pictures with the comments “my beautiful granddaughter Audrey!” or“my first grandchild!” The user subsequently “liked” Grandma's sharings.The curator, which monitors the user's Facebook account, upon notingthese activities, suggests that the user send a “Baby Brag Book”—avirtual picture product featuring Audrey's pictures—to Grandma forChristmas. This is accomplished by the curator presenting the virtualpicture product for the user to accept, reject, edit, or to ask forsimilar products.

The following are some examples of proactively engaged interactions bythe curator with the user:

EXAMPLE 1: “I've taken the liberty of creating a photo album of yourrecent Cruise to Alaska. You took some spectacular photos of whales andglaciers. Your Sister Betty likes Alaska and whales too, should I sharethe album?” . . .

EXAMPLE 2: “I've noticed that you took a picture of your lunch yesterdaywhen you were at, “Tokyo Sushi Bar” with your friend Jenny today. Youalso took a nice photo of Jenny smiling. Should I send those photos toJenny or post them to Facebook? I can also include a comment if youwould like. Just tell me what you would like it to read, I'm listening .. .

EXAMPLE 3: “I see you are scheduled to have lunch with your friend Jennyagain next week and I noticed that Jenny has a birthday next week. Wouldyou like me to make a birthday card for Jenny? I can use one of the manyphotos of Jenny and you together. Here are the best photos of Jenny andyou together, which one do you like? Jenny, often posts photos andarticles about “cats”, I can make a “Cat themed” birthday card for herif you would like.

EXAMPLE 4: “I see that you have a great portrait of your daughterBrenda, but the photo has a problem with “red eye”. I can correct it ifyou wish. Should I correct it?

EXAMPLE 5: “I have collected all of best shots of you and your sistersand made a collage. Mother's Day is in a week, would you like me to usethis collage to make and send a Mother's Day card for you Mom?

EXAMPLE 6: “Nice fireworks, Happy July 4th! What other holidays do youenjoy celebrating?”

EXAMPLE 7: “Another great soccer game photo! Do you know anyone on theteam”?

EXAMPLE 8: “I see you travelled to India. Do you have friends orrelatives there?”

Some examples of products that can be offered to the user out of his orher media assets:

Example 1: “A Year of Selfies”—When the number of “selfies” found amonga user's media assets exceeds a predetermined threshold, a virtualpicture album or slide show may be created. For example, to providebetter scene-to-scene transitions in a slide show of selfies, thecurator may (i) order the selfies chronologically, (ii) align the eyesor faces from picture to picture, (iii) process the pictures usingselective zooming, crop, rotating or other image editing techniques toestablish matching ocular separations or face orientations for theselfie images, and (iv) sequence the images to make a user-controlledvariable speed for the digital slide show. Examples of such techniquesmay be found, for example, in U.S. Pat. No. 8,692,940, entitled “METHODFOR PRODUCING A BLENDED VIDEO SEQUENCE” assigned to KODAK ALARIS®.

Example 2: “Selfie Flip Book or Album”, “Selfie Poster” (e.g., anarrangement of 5″×4″ and 4″×4″ selfies on a 24″×36″ poster, showing arange of user-selected expressions), a set of “Selfie Drink Coasters”,and“Selfie Stickers”

Example 3: Selfie or Portrait Auto Tool—A tool that includes utilitiesfor (i) aligning, zooming, panning, or rotating images anchored atcertain face feature points (e.g., eyes, ocular separation, mouth, ornose) that can be applied post-capture or as capture upon a trigger, and(ii) arranging images chronologically or through a set of recognized“range of expressions”, “gaze directions”, incremental head positions.These features allow creation of image collections that have seamlesstransition from frame-to-frame (gif, movie, VR model). Such a product isuseful, for example, for photographing small children and infants, or asa creative or automatic image creation tool.

To offer media-centric products to a user, curator may also takeadvantage of the user's activities and situations in real-time.Internet-of-things (IoT) devices—which include a large number ofnetworked sensors (e.g., thermostats, appliances, home automationsystems, and automotive devices)—are expected to reach 26-30 billion by2020. The curator can take advantage of such sensors. For example, ahome equipped with an IoT security system may keep track of theidentities of the persons in the house at all time, as the systemidentifies each person and assigns a unique security code when theperson enters the monitored perimeter. (A person's presence may bedetected from registration signals from his or her smartphones).Alternatively, at any given time, the heating or cooling system or thelighting may operate according to the preferences of particular peoplecurrently in the house. The curator may use such information to interactwith the people in the house (e.g. the curator can provide the curatedproduct to a specific person after the person arrived at his or her homeand has relaxed for a while). The virtual curator may also change itspersona based on the persons detected to be at home.

The sensor output signals of IoT-enabled appliances can indicate thetype of activities occurring in the house. For example, the time ofswitching on the television set or turning to a specific channel on atelevision set, when correlated with a program schedule, may indicate aspecific television program being watched. Similarly, frequent accessesto a refrigerator may indicate cooking. The operations of the coffeemachine, a washer and a dish washer may indicate coffee being brewed,laundry and dishes being cleaned, respectively. Using such data, thecurator can select a favorable time to interact with a given user. Forexample, the curator should refrain from interacting with a userwatching sports program on the television, or performing certaindomestic chores. Conversely, the curator should interact with a userduring an advertisement break, or between television programs. A patternof user behavior may be learned over time. The curator may maintain ahistory of its interaction with a user, to allow the curator to analyzewhen the user responded favorably, by viewing or buying the curatedproduct or engaging with the curator, or when the user did not respondfavorably. Based on the maintained interaction history and the detectedactivities in real time, the curator may learn rules to determine theideal conditions to engage the user.

Smart television sets and mobile devices are equipped with microphones,computational processors, and algorithms that may be used to monitoractivities in the environment. Such activities may include a person inthe room watching television or the person using voice commands, such as“mute”, “change channel”, “call Mom” and the like. The music that isbeing played in the room may suggest a user's mood. If a person isdetected to have just been involved in an argument with his or herspouse, based on the detected loudness and tone of their interaction andcertain keywords being recognized to have been spoken, the curatorshould not try to recommend purchase of pictures, at least notpurchasing pictures to give to the spouse! If a user is detected to havejust cleaned up a dog mess, the curator should not suggest framedpictures of the dog. In contrast, if snowy weather is detected, thecurator may suggest pictures of last winter's trip to Florida.

A wearable health or sport device (e.g., a FitBit device) monitors auser's movements and physiological conditions and can be used to predictthe user's mood. Monitored and recorded indications of physicalexertion, movement, heart rate, and breathing pattern may represent auser's current physical conditions that can be compared to knownprofiles to estimate or approximate the user's psychological conditionor mood.

Thus, using physiological, behavioral, and emotional data actively orpassively, whether the data is provided by the user, recorded bypersonal or environmental sensors, or interpreted from images, video, oraudio recordings (e.g., using such algorithms as Eulerian VideoMagnification or motion vector determination).

“Virtual reality” (VR) technology is increasingly important in consumerelectronics. Viewing photographs in an immersive environment iswell-known, such as illustrated by FLICKR® VR and similar applications.With a suitable VR viewer, which can be an integrated specialty device,or a headset frame with lenses to accommodate a large display smartphone, images and videos can be displayed in a VR environment. A userwearing a VR headset may be presented with a VR environment forphotograph browsing, for example, in the form of a virtual sphere aroundthe user. The images of a photo collection are projected asinward-facing images lining the surface of the virtual sphere. In thatexample, the user can navigate the VR environment using a pointingdevice, gestures recorded from the VR headset, VR gloves or controllers,or gestures recoded by one or more cameras monitoring the user, headmovements, facial expressions or voice commands. The user can sort andselect images, magnify images, and select presentation formats, such asa conventional sequential presentation typical of a photographic “slideshow” or presentation formats more suitable to VR environments, such asspiral or rotational presentation of images within the VR environment.The organized photograph collection can be displayed as static ordynamic visualizations using the metadata and tags associated withindividual images and video clips as sort and presentation criteria,such as recognized faces, objects, and scene types, time frames orlocations. In addition, an individual image or a set of images can beused to create the navigable environment. Also, original images recordedas stereoscopic images or images that have been converted tostereoscopic images will appear to have depth in a VR environment.

The curator can also serve as an educational resource, e.g., to help auser put together a report on collected images or videos, or assist theuser on a project that is based on the collected images and videos. Asmany users either do not like the sound of their own recorded voices orlack the confidence to narrate even a brief story, the curator canprovide audio narration for use with a multimedia storytelling productor feature. Generally, therefore, the curator may serve as a “userinterface shell,” i.e., an application that coordinates between theuser, image organization and manipulation software, and the user'scollection of personal content.

The curator may also process the media assets to facilitate sharing themedia assets among friends and on social networks. Techniques for suchprocessing may be found, for example, in U.S. Pat. No. 9,185,469,entitled “SUMMARIZING IMAGE COLLECTION USING A SOCIAL NETWORK”, toGallagher et al.

The curator may perform content consumption pattern analysis, whichprovides analytics data such as page views, time on site, bounce rates,ad clicks, and downloads. Such analytical data provides insight on wherethe users' primary unmet needs and interests that can be used to developpersonas for the users'.

The above detailed description is provided to illustrate the specificembodiments of the present invention and is not intended to be limiting.Numerous modifications and variations within the scope of the presentinvention are possible. The present invention is set forth in theaccompanying claims.

What is claimed is:
 1. A computer-implemented method for providing acustomized media-centric product, configured to run a networkedcomputational device comprising a processor and a storage device, themethod comprising: the processor receiving newly acquired media assetsfrom a user; the processor initiating an interactive curator forcreating a customized media-centric product; the interactive curatorreceiving permission from the user to access the user's metadata,wherein the user's metadata comprises the user's contacts and the user'sevent calendar; the processor accessing the user's metadata stored onthe networked computational device; the processor converting the user'smetadata into cross-referenced metadata, wherein the cross-referencedmetadata comprises new metadata created by the interactive curatorrepresenting at least one personal trait of the user; the processoranalyzing the cross-referenced metadata against a trigger set, whereinthe trigger set comprises trigger events; the processor identifying amatch between the cross-referenced metadata and a trigger event,precipitating the interactive curator to prepare a customizedmedia-centric product, wherein the customized media-centric product isincorporates at least one of the user's media assets or newly acquiredmultimedia assets; and presenting, via a graphical user interface on adisplay, a rendering of the customized media-centric product to theuser.
 2. The computer-implemented method of claim 1, wherein the userinteracts with the interactive curator via a networked computationaldevice selected from the group consisting of: a mobile device, a desktopdevice, a personal computer system, or a retail photo kiosk system. 3.The computer-implemented method of claim 1, wherein the trigger setcomprises temporal trigger events.
 4. The computer-implemented method ofclaim 3, wherein the temporal trigger events are specific datesidentified from network-connected users' calendars, social networkaccounts, posts, comments.
 5. The computer-implemented method of claim1, wherein the trigger set comprises location trigger events.
 6. Thecomputer-implemented method of claim 1, wherein the trigger setcomprises system trigger events.
 7. The computer-implemented method ofclaim 1, wherein the trigger events are adapted to accommodate differentcultural customs.
 8. The computer-implemented method of claim 1, furthercomprising the interactive curator engaging the user to acquireadditional information about the media assets to be incorporated in themedia-centric product.
 9. The computer-implemented method of claim 8,wherein the interactive curator engaging the user further comprisesacquiring relevance information regarding the media-centric product andthe trigger event.
 10. The computer-implemented method of claim 8,wherein the interactive curator engages the user graphically as anavatar via the graphical user interface and audibly via the networkedcomputational device.
 11. The computer-implemented method of claim 10further comprising the processor receiving a user selection of at leastof (i) a desired level of interaction with the interactive curator and(ii) interaction modality.
 12. The computer-implemented method of claim11, wherein the interaction modality comprises one or moreuser-selectable interaction modalities selected from the groupconsisting of: text message, email, voice message, and audio message.13. The computer-implemented method of claim 1, wherein the user'smetadata comprises information regarding reoccurring or spontaneous lifeevents.
 14. The computer-implemented method of claim 1, wherein theuser's metadata further comprises one or more of: (i) metadata recordedfrom contemporaneous activities by a wearable health device, and (ii)pulse rates and blood flow rates extracted from video using EulerianVideo Magnification.
 15. The computer-implemented method of claim 14,wherein the match between the cross-referenced metadata and the triggerevent is based on the user's metadata recorded from contemporaneousactivities by a wearable health device.
 16. The computer-implementedmethod of claim 1, wherein the customized media-centric product isprepared by a story generation the process comprising: (i) identifyingimportant media assets, (ii) ranking the important media assets, (iii)grouping the important media assets, and (iv) prioritizing the groups ofimportant media assets based on degrees of similarity between adjacentmedia assets in the group.
 17. A computer-implemented method forproviding a customized media-centric product, configured to run anetworked computational device comprising a processor, the methodcomprising: the processor receiving newly acquired media assets from auser; the processor initiating an interactive curator for creating acustomized media-centric product; the interactive curator receivingpermission from the user to access the user's metadata, wherein theuser's metadata comprises information derived from the user's socialnetwork activities related to each newly acquired media asset; theprocessor accessing the user's metadata and analyzing the metadataagainst a trigger set, wherein the trigger set comprises trigger events;the processor identifying a match between the metadata and a triggerevent, precipitating the interactive curator to engage with the user,wherein engaging with the user comprises proposing a customizedmedia-centric product to the user via a graphical user interface on adisplay and requesting user feedback, wherein the customized mediacentric product incorporates at least one of the newly acquired mediaassets from the user.
 18. The computer-implemented method of claim 17,wherein the interactive curator engages with the user graphically as anavatar via the graphical user interface and audibly via the networkedcomputational device.
 19. The computer-implemented method of claim 17,wherein engaging with the user further comprises providing an audionarration in association with the proposed customized media-centricproduct.
 20. The computer-implemented method of claim 17, whereinengaging with the user comprises offering to post the proposedcustomized media-centric product on the user's social networks.